Method of and apparatus for registration of a robot

ABSTRACT

In order to register a robot, the operative parts of the robot are placed in a first condition. An image acquiring arrangement carried by the robot is directed towards an item, and the image of that item is acquired and stored. Subsequently the component parts of the robot are moved so that the image acquiring arrangement can acquire a second different image of the item. The process is repeated to acquire data including a plurality of images and, for each image, data concerning the position of the component parts of the robot when the image was acquired.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method of and an apparatus forregistering a robot.

In particular, in one aspect, the invention relates to an apparatus forand a method for the registration of a robot to a work piece, to enablethe robot, or part of the robot, to be placed in a preciselypredetermined position relative to the work piece, thus enabling therobot to operate accurately on the work piece, and to perform desiredoperations at precisely predetermined points on the work piece.

In an alternative aspect, the invention relates to a method of and anapparatus for registering a robot in which the precise position andorientation of a tool carried by part of the robot is determinedrelative to that part of the robot, to enable the robot subsequently toposition the tool, or part of the tool, in a predetermined position.

2. Description of Related Art

In many situations where robots are used, the robot is programmed tooperate on a work piece of a precisely known size and shape. An exampleof such a situation is where a robot operates on a motor vehicleassembly line, where each work piece, in other words each vehicle thatthe robot has to operate upon, is of a precisely known size and shape.In such a situation the robot can be pre-programmed to carry out asequence of moves which are appropriate for the task that the robot hasto perform.

There are, however, various situations where a robot has to performtasks on a work piece where the size and shape and other characteristicsof the work piece are known approximately, but the precise detailsdiffer from specimen to specimen. Examples include hand-made items,items made of semi-rigid or deformable material and, in particular,living tissue, for example in a situation where the robot is used in anoperating room. In such a situation, it it not uncommon for the task ofthe robot to involve the steps of penetrating the “work piece” in orderto access a particular internal target or pathway. Typically, theinternal target or pathway is invisible from the surface of the workpiece, and, especially in the situation of a robot acting on a humanpatient in an operating room, it is essential that the robot access theinternal target or pathway accurately.

A convenient method which has been used for specifying appropriatecoordinates and instructions to the robot for these types of tasksinvolves the use of an image guided technique. In utilizing thistechnique, an image is acquired of the work piece, for example by usingx-rays, magnetic resonance imaging, ultra-sound imaging or some othercorresponding technique. The imaging technique that is utilized ischosen so that the internal target or pathway is revealed or can bedetermined.

The image is computer-processed and displayed in a form that isconvenient for a human operator. Depending upon the preference of theoperator, and the nature of the internal target or pathway, the displaymay be presented as a series of “slices” through the work piece, or asthree orthogonal views through a designated point, or alternatively as athree-dimensional reconstruction. There are many types of imageprocessing algorithms available for this purpose.

Using a pointing device, such as a mouse, the human operator can nowspecify on the computer representation of the work piece where thetarget or pathway is located. The operator may also indicate therequired approach path for the robot to reach the target or pathway,from the surface of the work piece. The coordinates of the key points ofthe desired approach path and also the target or pathway are readilydeterminable from the pixel or voxel which the operator has specifiedwith the pointing device. The coordinates can easily be converted into aseries of instructions which can be passed to the control arrangement ofthe robot so that the robot effects the appropriate moves to cause theend effector carried by the robot to follow the desired approach path tothe target or pathway.

The robot may be positioned above or adjacent the real work piece, butbefore it can carry out the instructions provided to the robotcontroller, a registration process must be performed to bring the framesof reference of the robot, the work piece and the computerrepresentation into alignment. In other words, the robot must initiallybe positioned in such a way that when the robot carries out theinstructions, which have been passed to the robot control arrangement,the end effector which is moved by the robot follows the appropriatepath relative to the work piece.

It has been proposed (see EP-A-0,755,660) to bring the frames ofreference of a robot, a work piece, a computer control arrangement intoalignment by utilizing markers which are visible, both to the eye and tothe imaging technique which is utilized the markers being fixedtemporarily to the work piece before imaging is effected. To effectregistration between the robot and the work piece, the computer mouse isused to indicate a first marker on the computer-displayed image, and therobot is then moved so that the appropriate part of the end effectorcarried by the robot touches the corresponding marker on the real workpiece. Thus, effectively, the end effector of the work piece is moved toa predetermined position, which is a known position in the computergenerated image. This process is repeated with the other markers. Arotation and translation can then be specified which brings thecoordinate frame used by the robot into registration with the coordinateframe of the work piece as displayed in the computer generated image. Atleast three markers are required, but in practice better results areobtained by using between five and ten markers, and also by taking twoor three readings at each marker. A “least squares” approach may then beused to manipulate the data to obtain a best fit.

This technique is effective, but is tedious for the operator, and proneto inaccuracy due to errors in positioning either the robot or themouse.

Thus, one aspect of the present invention seeks to provide a method forregistering a work piece to a computer model which does not possess thedisadvantages of the prior art.

In some types of robot tasks, the robot is required to handle more thanone tool, or may handle a tool that may possibly have been deformed inshape, or which may have a variable length. For example, the tool partof the tool may be worn away. In many situations accurate positioning ofthe tip of the tool is required and consequently, it is necessary toregister or “calibrate” the robot so that the position of the tip of thetool, relative to some fixed point on the robot, is known precisely.This will enable the robot to position the tip of the tool in therequired position, regardless of the identity of the tool that is beingused, or regardless of any deformation that may have occurred to thetool. Thus, in another aspect, the present invention provides a methodfor achieving such a registration or “calibration ” without the need totouch the tip of the tool. This is important in applications where thetip of the tool may easily be damaged or where it is essential that thetip of the tool be maintained in a sterile condition.

BRIEF SUMMARY OF THE INVENTION

According to this invention there is provided a method of registering arobot relative to a work piece which has a plurality of visible markers,wherein the robot carries a visual image acquiring arrangement, saidmethod comprising the initial step of acquiring images of the work pieceand markers, and storing the images, and preparing commands for therobot in terms of the coordinates of the stored images, and comprisingthe subsequent steps of determining data by placing the operative partsof the robot in a first condition with the visual image acquiringarrangement directed towards the work piece, acquiring and storing animage of the work piece and markers and subsequently moving thecomponent parts of the robot so that the image acquiring arrangementsacquires a second, different image of the work piece and markers, andrepeating the process to acquire data comprising a plurality of imagesand, for each image, data concerning the position of the component partsof the robot when the image was acquired, said data being stored. Themethod further comprises the step of comparing the initially acquiredimages of the work piece with the images acquired by the visual imageacquiring arrangement in the said subsequent steps, and effectingregistration of the robot to enable the robot to perform one or moreoperations with reference to the work piece on the basis of the commandsprepared for the robot in terms of the coordinates of the stored imagesacquired in said image or step.

Advantageously there are four or more visible markers provided on thework piece.

In one embodiment the method may include subsequent steps for thecalibration of a tool held by the robot. The method comprises the stepsof using the image acquiring apparatus to acquire successive images of atool carried by part of the robot, when that part of the robot is indifferent positions, and subsequently determining the precise relativeposition of the tip of the tool, relative to the part of the robot.

The invention also relates to an apparatus for registration of a robotrelative to a work piece. The apparatus comprises visual image acquiringmeans, and means to mount the visual image acquiring means on part of arobot, means to store an image acquired by the visual image acquiringmeans, and means to store a subsequent image acquired by the visualimage acquiring means. The apparatus further comprises means fordetermining and storing data concerning the position of at least onecomponent part of the robot when each image was acquired, and whereinthe apparatus further comprises means to store a plurality of previouslyacquired images of the work piece, and means for comparing the imagesacquired by the visual image acquiring means with the previouslyacquired images, and means to generate control signals adapted tocontrol movement of the robot to effect registration of the robotrelative to the work piece.

Conveniently the image acquiring means is mounted on a robot and islocated to acquire an image of a tool carried by part of the robot whenthat part of the robot is in different positions.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

In order that the invention may be more readily understood, and so thatfurther features thereof may be appreciated, the invention will now bedescribed, by way of example, with reference to the accompanyingdrawings in which

FIG. 1 is a diagrammatic view illustrating the acquisition of an imageof a work piece, and the subsequent processing of the image;

FIG. 2 is a diagrammatic view illustrating a robot and the work piece;

FIG. 3 is a block diagram illustrating the control arrangement for therobot of FIG. 3;

FIG. 4 is a schematic view illustrating a part of a robot carrying anend effector in the form of a tool;

FIG. 5 is a block diagram.

DETAILED DESCRIPTION OF THE INVENTION

Referring initially to FIG. 1 of the accompanying drawings a work piece,which may comprise an inanimate object, or which may comprise an animateform 1 that is, for example, to undergo surgery, is provided with aplurality of visible markers 2. It is preferred at least three markersare provided, but preferably a greater number of markers is present. Ifonly three markers are provided, or if a greater number of markers areprovided, the markers should not be located at positions equivalent tothe corners of a regular geometric figure, such as a equilateraltriangle or a square, but instead should be irregularly positioned.

As shown in FIG. 1, an image acquiring arrangement 3 is utilized toacquire an appropriate image of the work piece and the markers. Themarkers are selected to be “visible” to the image acquiring arrangement3. The image acquiring arrangement 3 may take any convenient form, andmay thus comprise an x-ray arrangement, an ultra-sonic scanningarrangement, a magnetic resonance imaging arrangement or otherequivalent arrangement capable of acquiring an appropriate image. Theimage is processed 4 and stored 5. The stored image may be utilized byan operator to prepare commands which are subsequently to be followed bya robot. The commands may specify a path to be followed by a toolcarried by the computer, relative to the work piece, or may specify aninternal target to be operated on by the robot, for example, if therobot is to control the firing of a particle beam or a narrow beam ofradiation to a particular site within the work piece.

The commands 6 and the stored image are retained in an appropriateformat.

FIG. 2 illustrates diagrammatically a robot 7 which is located adjacentthe work piece 1 having the markers 2 in position.

The robot 7 comprises a housing 8 carrying a support column 9, to whichis pivotally connected a support arm 10 which carries an end effectorcarrying arm 11. The end effector carrying arm 11 carries, at its freeend, a tool or other end effector 12. The various arms areinter-connected by pivotal connections.

Mounted on the end effector carrying arm 11 is a camera or visual imageacquiring apparatus 13 in the form of a television camera, a digitalcamera, CCD device or the like. The camera 13 is adapted to acquire avisual image of the work piece 1 and the markers 2.

Referring now to FIG. 3, the camera 13 acquires an image of the workpiece, together with the visible markers, and the image is digitized andfed to a store 14. The store 14 also stores data relating to theposition of the components of the robot. All of the various movableparts of the robot are moved under the control of a sophisticatedcontrol arrangement, and consequently data can readily be generated, 15,indicating the precise position of all the component parts of the robot.

Once a first image has been stored, the robot is moved, and a secondimage is taken. This process is repeated several times.

The stored images in the store 14 are compared, in a comparator 16, withthe stored images from the store 5 shown in FIG. 1. On comparing thestored images, a registration procedure 17 is carried out in which theposition of the camera 13 carried by the robot is determined with regardto the frame of reference identified by the stored images of the workpiece.

Consequently, the control arrangement 18 of the robot can determine,with regard to the coordinate frame of reference of the robot, theprecise position and orientation of the work piece relative to therobot, thus enabling the robot to follow accurately commands 19 whichwere derived with reference to the original images of the work piece.

It is thus to be appreciated that the robot may effect preciselypredetermined operations on the work piece without any time consumingpre-registration relative to the work piece, using an automatedarrangement which minimizes the risk of errors arising.

Referring now to FIGS. 4 and 5 of the accompanying drawings, FIG. 4illustrates schematically an end effector carrying arm 20 of a robot onwhich is mounted a camera image acquiring device 21, such as atelevision camera, digital camera or CCD device. The camera 21 is solocated that, within the view of the camera, there is a tool 22 which iscarried by the end effector carrying arm 20.

It is desirable for the control arrangement within the robot to havedata indicating the precise position of the tip of the tool 22 withregard to the end effector arm 20, or some other part of the robot. Inorder to acquire this data, the tip of the tool 22 is viewed with thecamera 21. The location of the pixel of the tip of the tool is effectedusing an appropriate thresholding technique and computer analysis of themeasured image. It is to be appreciated that when viewing the tip of thetool, it may be appropriate for a “background ” of a contrastingbrightness or color to be located immediately behind the tool as viewedby the camera. This will facilitate the thresholding method utilized bythe computer to identify the pixel in which the tip is located.

Data is stored relating to the position of the end effector carrier andthe pixel location of the tip of the tool.

Subsequently, the end effector carrier is moved, and the process ofviewing the tip of the tool, determining the pixel location of the tipof the tool, and the storing of data relating to the position of the endeffector carrier, and the pixel location of the tip of the tool isrepeated. This process may be repeated many times, until sufficient datahas been accumulated for an accurate calculation to be performed todetermine the position of the tip of the tool relative to the endeffector carrier. Theoretically, it is possible to perform thecalculation when three sets of data have been taken. However, greateraccuracy can be achieved if a greater number of sets of data are takenand stored.

I claim:
 1. A method of registering a robot relative to a work piecewhich has a plurality of visible markers, wherein the robot carries avisual image acquiring arrangement, said method comprising the initialstep of acquiring images of the work piece and markers, and storing theimages, and preparing commands for the robot in terms of the coordinatesof the stored images, and further comprising the subsequent steps ofdetermining data by placing operative parts of the robot in a firstcondition with the visual image acquiring arrangement directed towardsthe work piece, acquiring and storing an image of the work piece andmarkers and subsequently moving component parts of the robot so that theimage acquiring arrangement acquires a second, different image of thework piece and markers, and repeating said steps to acquire datacomprising a plurality of images and, for each image, data concerningthe position of the component parts of the robot when the image wasacquired, said data being stored, the method further comprising the stepof comparing the initially acquired images of the work piece with theimages acquired by the visual image acquiring arrangement in the saidsubsequent steps, and affecting registration of the robot to enable therobot to perform one or more operations with reference to the work pieceon the basis of the said commands prepared for the robot in terms of thecoordinates of the stored images acquired in said image or step.
 2. Amethod according to claim 1 wherein there are four or more visiblemarkers provided on the work piece.
 3. A method according to claim 1futher comprising subsequent steps for the calibration of a tool held bya robot, the subsequent steps comprising the steps of using the imageacquiring apparatus to acquire successive images of a tool carried bypart of the robot, when that part of the robot is in differentpositions, and subsequently determining the precise relative position ofthe tip of the tool, relative to said part of the robot.
 4. An apparatusfor registration of a robot relative to a work piece comprising visualimage acquiring means, and means to mount the visual image acquiringmeans on part of a robot, means to store an image acquired by the visualimage acquiring means, and means to store a subsequent image acquired bythe visual image acquiring means, said apparatus further comprisingmeans for determining and storing data concerning the position of atleast one component part of the robot when each image was acquired, andwherein the apparatus further comprises means to store a plurality ofpreviously acquired images of the work piece, and means for comparingthe images acquired by the visual image acquiring means with thepreviously acquired images, and means to generate control signalsadapted to control movement of the robot to effect registration of therobot relative to the work piece.
 5. An apparatus according to claim 4wherein the image acquiring means is mounted on a robot and is locatedto acquire an image of a tool carried by part of the robot when thatpart of the robot is in different positions.